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This paper describes a framework to facilitate the authoring of sound with spatio-temporal relativity in virtual or 
augmented environments. It is based upon the premise that the interactor with the environment creates their own 
individual interpretation of 'narrative': as a result of their movement through the space, accumulating fragments of 
stimuli or content to create meaning. Inspiration is gained from the actions of the archaeologist, using stratigraphy as a 
recording practice, that records spacio-temporal paradigms. As the author of a spatialised soundscape composes a 
database of narrative fragments for the user to encounter, the paper proposes a distinct design process that reverses the 
excavation procedure, and re-imagines stratigraphical layers as phases of sound in the present tense. 


INTRODUCTION 

The ambition to create immersive audio-augmented 
environments raises issues of time in a spatial 
environment. This is particularly the case when sound 
samples are presented as context-aware, or as part of a 
narrative structure, referencing location position [1]. As 
the user moves and experiences the space, each 
'triggered' sound plays, according to a combination of 
location data and reference to indexes representing user 
activity. During the period of time that the environment 
is occupied, a spatial data history, or memory, 
accumulates representing the interactive movements of 
the user - all that must be anticipated in advance by the 
creator/programmer of the immersive environment. 

In consideration of the design process, the spatial 
positioning of sound sources within a XYZ coordinate 
system is relatively straightforward, using established 
graphical mapping techniques, applied in a computer¬ 
generated representation of the space. Yet 'history' 
occupies a further dimension, that of time, dividing the 
continuum of experience into different stages and 
events. 

The impetus for this investigation was born in issues 
discussed during guest participation within the First 
LISTEN Expert's workshop [See http://listen.imk.fhg.de 
for general information on the project]. The following 
paper, as a distillation of this author's MSc thesis [2], 
introduces a process that assists the design and 
management of sound temporality or relativity within a 
3D space, exploring the component stages. 


1 PRELUDE 

The dialectical nature of agency and narrative within 
interactive environments, has promoted structuralist- 
informed theory to understand how meaning is formed 
in such spaces. Referring to role-playing-game 
environments, it has been suggested that the 
transformation of basic consequential and consecutive 
narrative units into meaning, and therefore a sense of 
overview, is the result of "conscious work" in virtual 
environments by the nature of participation within it 

[3] . Or in other words, the interactor creates their own 
individual narrative from their own agency or activity 

[4] . 

In the scenario of an audio-focused virtual or augmented 
environment, the interpretation of the experience would 
depend upon the function of the content, be it 
informative or creative. For example an artistic 
application may use a dynamic combination of 
descriptive tropes, such as metaphor, metonymy and 
synecdoche, via spoken word, poetic association of 
noise, sound effect, and emotions from different musical 
elements. Immersed within a changing perceptual field, 
a constantly fluctuating semiotic association between 
sounds, environment, and spatial arrangement may 
develop. As the participant navigates through the space 
according to their attraction and mood, an individual 
accumulation of experience would evolve and be 
translated into a personalised soundscape. Thus the 
interactor creates their own constituted meaning of the 
work (consider [5] as a audio/visual example). 
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2 ARCHEOLOGICAL RE-PRESENTATIONS 

The interpretation of a virtual or augmented site, based 
upon the exploration of the environment and the 
fragments of narrative encountered, suggests some 
metaphorical parallels to the action of the archaeologist, 
likewise charting and gathering items of interest from an 
excavation. The following sections are informed by, 
and gain inspiration from, the recording processes of 
archaeological stratigraphy, and explore its relevance in 
assisting creativity and the information management of 
time-relative audio-augmented projects. 

2.1 The archeological record 

Beyond the unique spatio-temporal aspect of its data, 
archaeology can be approached as the understanding of 
the past, through the representational metaphor of an 
archaeological record, treated as an independent and 
material representation of certain events and processes. 
Note it is normally asserted that archaeology can only 
study those aspects of the past that are capable of 
material representation in this record [6]. 

The purpose of the record satisfies three aspects for 
analysis. Firstly it transcribes the material (remains) in 
order to produce an archive of representations, 
consisting of a variety of textual comments, images and 
material samples. The representation is of a solely 
empirical nature, and in modern times accuracy has 
been of great importance. Secondly, the material record 
represents past events and processes at the site in 
question, analysing the material origins and the 
processes that have had an effect since, although the 
dynamics of change are seen as too ambiguous to offer 
observational evidence, and so are not part of the report. 
Lastly, it analyses the meaning of events of the record in 
the terms of historical process. In other-words, the 
remains are inadvertently a representation of the 
historical process, as they are the consequences of it, 
and can be interpreted as the basis of indication for the 
activities that may have taken place at the site of record. 
[ 6 ], 

The creation of the archaeological record, and its stages 
of development, forms an archival process that begins, 
after a period of research about the site, with an 
excavation. Documenting with scientific precision, 
modern archaeology records the spatial dimensions and 
stratigraphic layers of the excavation site using plan and 
section diagrams, plus a detailed account of finds 
(artefacts) with photographs and illustrations. The 
following stage aims, using a relativity diagram called 
the stratigraphic sequence and radiocarbon dating of 
suitable artefacts, to create a time sequence of the site, 
identifying 'phases'. This sequence then is subject to 


comparative research and analysis, of which the results 
are written into an excavation document or report. 
Usually this process ends with the archival of the 
portable finds, accompanied by a published report, with 
a public body, such as a museum [7]. 

The traditional process of an archaeologist, in summary, 
follows a process in which the details are recorded in 
2D plans* and sequence diagrams, their relativity 
gauged and chronology investigated, resulting with an 
inteipretation of the history of the occupied site. A 
narrative is supposed from the combined interpretation 
of fragmentary artefacts over a period of time. 

*See [8,9] as examples of recent research into new VR / 
context-aware archaeological systems. 

2.2 Reversed process 

This paper proposes that the user of an interactive 
virtual or augmented environment acts in a similar 
fashion, exploring for, finding, and interpreting virtual 
'artefacts' of interest, which may lead to an 
understanding of the experience. If the author is 
creating an environment for the interactor to experience, 
through their occupation over time, a design 
methodology such as the following - a reversal of the 
creation of an archaeological record - suggests itself as 
being a useful guidance (see fig. 1). 

> Experience or function 
of virtual/augmented 
environment. 

> Synopsis (history). Genre, 
Style, Narrative elements, 
Absent/Present Subject? 

> Inter-related loop which 
defines distinct phases/ 
periods/chapters of 
development. Identify 
'artefactual' sound sources 
as historical, cultural and 
environmental clues to 
narrative. 

> Record and chart all 
sound, indicating 
spatial boundaries, for 
each phase in 3D space. 

Figure 1: The reversed archaeological process. 
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As 'interpretation', 'documentation' and 'recording' (data 
spatially) are all familiar processes within sound design 
for 3D computer spaces, the following section will 
elaborate upon the definition of 'phases' and how it may 
be applied for the afore-mentioned purpose. 

2.3 Stratigraphy and phasing of content 

Due to the destructive, irreversible approach of 
investigation, the removed deposits of an excavation 
dig, and whatever contents found within each context, 
are recorded empirically as the first step in the process 
of creating an archaeological record of a particular site. 

Using stylized section and plan diagrams to illustrate 
boundary contours, the spatial limits of each unit of 
stratification in both the horizontal and vertical 
dimensions are defined. Hard drawn edges mark the 
distinction between one deposit and another, and 
include layer numbers, numbered as each different 
stratum is encountered (fig. 2). 



Figure 2: Example of stratigraphical plan diagram 
after [7]. 


As the spatial boundaries of deposits excavated are 
mapped, so also are the relationships and interfaces 
between the deposits in consequence of their extraction 
[7]. The resultant abstraction is illustrated on the right 
of figure 2, as a Harris Matrix - the units encountered 
first are positioned at the top, being the most recent 
deposits in history, and the lowest represents the 
earliest. Each unit is positioned according to its relative 
relationship to its neighbour, so that in total there are 5 
possible states: Before, After, Contemporary, 

Equivalent and No-relation (or no determinable 
relativity). Hence Fig. 2 describes unit 9 as the earliest 
(and first) unit in the stratigraphy, followed by 8, and 
later by 7 which is the temporal contemporary of 6, and 
so on, so that 1 and 2 are the most recent stratigraphic 
units, positioned above 3. 


On the basis of this relativity, via date analysis or cross- 
examination of found artefacts within the stratum, the 
stratigraphic units illustrated in the Harris Matrix are 
divided into groupings or phases. These phases may 
represent the interpretation of distinct structural 
development, activity use within the space, or periods of 
narrative history, depending upon the evidence present. 

When the end of one phase, and the start of another, is 
identified, essentially an occupied surface is defined, 
which has for whatever reasons been abandoned, built 
upon, or adapted, and lies below the present surface. 
Therefore each phase has been occupied at different 
depths (and so at different times) in the history of the 
site. The history of the site can stretch over long 
periods of time, often hundreds and thousands of years. 

Of course, virtual and augmented environments are not 
designed to be occupied for such a timescale. However, 
the principle of divisible periods of occupation within 
the space can be applied. Regardless of what each 
phase may represent, be it relative narrative time - a 
consequential ordering of events - or continuous real 
time [3], the interactor's continued presence within, and 
so occupation of, the environment may be managed as 
relative or context units, and 'bundled' as phases. From 
the author or sound designer's point of view, the phases 
may be conceptualized, not as depths in layers at the 
site, but as successively changing 'overlays' (fig. 3). 



Figure 3: Conceptualizing augmented sound as 
consequential and consecutive overlying phases. 

Imagine that the stratigraphic sequence of sound passes 
down through the virtual or augmented site, so that the 
earliest phase (and its associated sounds) is initially 
synchronized with the first occasion of occupation. 
Thus through the progression of the stratigraphic 
sequence, the most recent section disappears, 
conceptually downwards, to be replaced by the sounds 
associated with the following phase, and so on, in a 
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manner similar to the space-time cube system used in 
temporal GIS systems [10]. Further, the conception of 
the layers may be elaborated, maintaining the metaphor 
of stratigraphy, by using the Harris Matrix diagram to 
assist the authoring of the sequential and consecutive 
detailed units of the sound design. 

2.4 Agency in an archaeological virtual space 

Allowing the interactor a sense of agency - that their 
actions affect and shape the environment they are 
present within - appears at odds with, and strains the 
usage of, the stratigraphical record as a useful metaphor 
in virtual and augmented environments. 

Due to the archaeological record traditionally being an 
empirical construct, it has a conceptual base that lies on 
what has been found - the result of human activity. 
However, current debate within archaeological theory 
seriously questions this presumption: [11] states that if 
the material residues (recorded in the archaeological 
record) are treated as representing agency, then the 
concept of a situated activity over time and space, in- 
other-words inhabitation , is lost. 

How is this debate relevant for sound design or 
engineering of virtual and augmented environments? It 
is the interactive-media author/coder's task to pre-empt, 
and plan the narrative potential of the virtual or 
augmented environment. As the above statement 
suggests: if the archaeological record is utilised strictly 
as a metaphor to construct an audio structure, although 
it may be dynamic in perception - as the interactor 
moves through space and time - it is essentially static in 
behaviour and location. The author has, in effect, 
planned the result of agency within the environment. 
Indeed there is a danger of re-grading interactive 
potential to a passive experience. In consideration of 
'situated activity', the record or data structure would be 
better approached, "partly as medium, and partly as an 
outcome of agency's existence" [11]. The implication 
then is that structures should not be simply regarded as 
constraining or restricting, rather they should be 
perceived as a field of possibility, reproduced by the 
practices that occupy that field. 

This philosophy may be applied, for example, as 
location or context-aware trigger areas (collision 
volumes), which facilitate change in non-sequential 
content development, moving forward, reversing, or 
even 'jumping' between phases [2]. Further, if the 
system consulted an index of visits the interactor made 
to each position, or programmable behaviours (e.g. AI) 
are attributed to the virtual sound-object as a 
consequence of the interactor's agency, complex 
narrative potential [3] may be possible. 


3 APPLICATION 

3.1 Content 

The above process, and the re-presentation of 
archaeological method applicable to sound, was applied 
in the construction of a creative audio-augmented 
virtual environment, entitled ' Garden Monologue ', 
consisting of spoken word and sound effect samples [2]. 
The content can be interpreted as a spatialised poem, 
presented in layers of constructed 'narrative', and aims 
to be an intermediate stage illustrating the creative 
potential of augmenting space with sound or spoken 
word. 

Spoken word, in particular, was chosen as sound content 
because it offered the greatest challenge to consecutive 
and consequential aspects of narrative. The intention 
was to put the methodology into real practice, to analyse 
its effectiveness and flexibility for application, its 
limitations, and elaborate upon unconsidered problems 
relevant specifically to the creative process of virtual 
environment design. As a test environment, the 
Unreal™ engine [12] and editor was used to design the 
environment, and facilitate, via custom-scripted code, 
the stratigraphical management of the sound. 

3.2 Relativity 

An independent Harris Matrix constructing program 
(ArchEd [13]) was used to assist the design process and 
relative layout of the sound samples. This program is 
normally used by archaeologists to assist overview of 
the various strands of relativity in context layers. 
However in this case, it offers a flexible mapping 
system which can assign relationships - earlier- than, 
later- than, contemporary, equal - between narrative 
elements and sounds as they would be experienced 
through the passage (or inhabitation) of time in the 
environment (fig. 4). The most obvious disadvantage of 
the tool for the described purpose in this paper is the 
limitation to chronological arrangement in 2D rather 
than 3D space. 

Once the whole arrangement was placed to a suitable 
degree, the sequence was then assessed to determine 
which samples might be grouped together into data lists 
of sounds at the same position (fig 5.). This part of the 
process mimics the creation of textual recorded or 
archival information gathered during the excavation of 
an archaeological site. It also acts as a transitory 
archive to be applied into the virtual or augmented 
environment: bundled together as a virtual sound-object 
to be activated by the interactor's agency. 
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Figure 4: Content arranged into relative temporal 
sequences using the ArchEd Harris Matrix program. 


other, a corresponding relativity index and time/pause 
counter: 

struct StructSound { 

sound sound;//sample sound 

byte volume;//0 -> 255 value 

float radius;//heard radius of sound 

float pitch;//normal = 1.0 

bool bLoop;// loop or play once? 

}; 

StructSound SSound[x]; 

//Stratigraphical sequence of x num. of sounds 


struct Relativity { 

int phase; //phase object exists within 

float pause; 

//length of time between samples, 

//used during consecutive (rollover) state 

}; 



Figure 5: Transferring relativity to data lists for 
application in sound-object arrays. 


3.3 Object data structure 

The Unreal™ engine can be adapted and enhanced 
using the proprietary language, UnrealScript, which is 
similar to a combination of Java and C++. A custom- 
coded object-orientated class - ArchaeSoundTrigger - 
was scripted as location-aware with two possible states, 
either 'one-shot' i.e. plays once, or 'rollover' i.e. follows 
through a sequence. These states facilitated both 
consequential and consecutive dynamics, respectfully, 
in the soundscape. This virtual sound-object is placed 
in a 3D position within the virtual or augmented space, 
and upon the interactor entering its location collision 
radius, plays sounds according to its state. The 
ArchaeSoundTrigger class essentially consists of two 
related data arrays - one array holds a rack or sequence 
of sound samples, each with its own feature data such as 
volume, collision radius, loop/single play etc; while the 


Relativity SRelate[x]; 

//Array that co-relates to each sound (x) in 
the SSound array. 

int PhaseNow; 
int CurrentSound; 

As the interactor enters the collision radius, the class 
checks if the current phase index, PhaseNow (initially 0, 
and incrementing 1,2,3 etc. upwards to a pre-defined 
limit) matches the 'exists in phase' value of the current 
position in the sound relativity data-list (also initially 0 
-> x). If true then the object plays the sound and 
depending upon the state, increments the counter 
CurrentSound to the next sound in the data array 
ssound, and the checking process is repeated again until 
all sounds in the sound object are played accordingly. 

Another virtual object class was scripted, Phasechange, 
to be placed within the augmented space. If the 
collision radius of this object were entered, a change in 
the development of the phase narrative, i.e. forward, 
backwards or to a particular phase, would be triggered. 
At the time of writing, a process that manages the 
passing of continuous real time, through the phases in a 
linear fashion, has not been fully developed. 

3.4 Sound design 

Once development of the sound-object characteristics 
were complete, and in conjunction with the relativity 
data-lists mentioned previously, the matter of 
positioning the sound-objects spatially began. Using 
the combination of plan and 3D perspective view - to 
most easily associate sound-object with spatial position 
- it became clear that issues regarding 'agency' and 
'archaeological record' also relates to the process of the 
sound design and the reversed archaeological 
methodology self-styled above. The record or 
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sequence, applied for creative or narrative purposes is, 
due to its fabrication, free from the confines of 
empirical representation. Flexible and adaptable to 
change, the record, in this case, is reflective of the 
creative potential of the digital medium for construction, 
the nature of the authoring tool, and therefore also 
process. Creativity and design, especially in 
experimentation, often relies upon taking steps 
backwards, revising and adapting to experience. 

It should therefore be presumed that the process of 
'recording' in the design process set forth in this paper is 
not absolutist, and may actually be better described as a 
period of 'transference and modification', according to 
run-time test experience in developing the environment. 
For example, during the process of designing 'Garden 
Monologue', both content and stratigraphic relativity 
were adapted during the transference stages of creating 
the sound data-lists, after sequencing. Responding to 
the experience of listening to interactive tests of the 
soundscape, variables such as radius, and sound samples 
were further 'tweaked', and either the matrix sequence or 
data list record, updating to suit. 

The resultant virtual or augmented environment, if the 
sound objects were visible, would illustrate a space 
populated by objects. Although they may not play until 
the phase matches the index of the first sound in the 
object's data array, they are positioned within the same 
field of navigation (fig. 6). 



Figure 6: 'Garden Monologue' virtual environment 
showing visible sound objects with existence index. 


4 ANALYSIS 

The success criteria of the custom-built virtual sound 
objects was to allow both consecutive and consequential 
dynamism of narrative, in other-words linear 
sequencing of sounds and non-linear interactivity of 
change. 


As basic information structures, the ssound and 
SRelate data arrays of the ArchaeSoundTrigger object 
class satisfied the transference of relevant information 
characterising the sound from the data list. The class 
successfully offered a mixture of options to define 
characteristics to a series of sounds, and play them in 
real-time, dependent upon the current phase and the 
interactor's trigger of the collision radius. The different 
options available to the designer - looped, one-shot, 
rollover, sequential or once only - implemented in the 
'Garden Monologue' example - suggests the possible 
richness of sound collage or narrative that may be 
accumulated and layered. 

A large part of the design process involves 
understanding and moving between different 'phase' 
plans, considering how agency and sound fragments 
may accumulate to form meaning. As each 
ArchaeSoundTrigger object was allocated to a group 
representing an origin of existence phase, it is possible 
to understand where within the temporal sequence the 
sound will be first triggered and so experienced. 
However, as some objects have a sound data-list which 
contains a combination of different assigned phases, it 
therefore means that those sounds which don’t originate 
in the assigned 'base' phase are lost to sight on a plan 
view of each phase. It is this author's opinion that 
further work is necessary to develop a more suitable 
visual interface if a graphical design system like 
Unreal™ is used. Of course if the design interface with 
the computer is within a coding environment, then 
hierarchal data-lists are most suitable. 

The Phasechange class, an object of agency and 
consequence, operates successfully to change the 
current phase counter in each temporal actor. However, 
an attached sound (which plays when triggered) proves 
only to be a useful indicator of change if the sounds, or 
spoken word/s, used are consistant or markedly different 
from the others, so that the interactor learns to associate 
the potential change of soundscape with that sound. 
However, the notion of an audio interface indicating 
such abstracted agency is problematic and deserves 
further development or better solution, as it is not 
presently satisfactory. 

This object in particular embodies the apparent 
disparities between linearity - archaeology or history - 
and non-linearity, agency and individual construction of 
narrative. It is possible, depending upon the assigned 
state, to jump from the beginning phase to middle, back 
and forward to the end phase of the sound data list, or 
even perpetually revolve through a loop of phases with 
little understanding of the full content. There is a danger 
that the interactor becomes frustrated with this situation, 
or indeed that they trigger all the 'one-shot' sounds, 
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leaving few or no sounds left to interact with, and an 
emphasised artificial environment that begs to be exited. 

These scenarios posed the question - should the 
temporal dimension (in this case, 'phases') also be 
controlled by an overriding real-time variable? With the 
experience of producing spatio-temporal sound in 
virtual space, reinforced by contemporary theories of 
archaeology, it is a claim of this paper that: to realise a 
sense of inhabitation it is necessary for both consecutive 
and consequential temporal structures to operate. 

During this investigation a serious limitation with the 
applicableness of the stratigraphical method to sound, 
was highlighted. Namely, the present system as it 
stands treats the sound to be played as an instance, with 
one node of existence, rather than what is essentially an 
entity with two nodes, i.e. a beginning and an end. The 
inclusion of a second relational node - the terminal point 
of the sound - would involve additional permutations of 
contextual relationships. For example beyond earlier- 
than, later- than, and contemporary, it would be 
necessary to have defining categories which considered 
asymmetrical synchronisation and potential 
relationships between the beginning and end of two 
different sounds. 

During this author's initial research into archaeological 
methods, one source [14] considered the problem of 
temporal structures in medieval settlements, proposing 
categories that defined such relationships in light of 
ambiguous overlapping temporal evidence of 
occupation at a site. The approach aimed to identify 
relationships that would be useful for interpretative 
constructs and subjective evaluation. This author 
decided that it was important to lay foundations first 
using [7]'s rather more objective approach. Suitably this 
limited implementation has highlighted the expressed 
need to further address the temporal occupation of 
sound, and its relationships of relativity to other sounds 
that also have a temporal occupation of time. 

Last, but not least, there appears to be a need to develop 
an artefact metaphor for non-interactive sounds, to 
compliment agency, which exist solely in a set time- 
frame of existence. Currently all relativity discussed in 
this paper has been based upon 'context' relationships. 
Within the field of computer-human interaction, the 
metaphor of 'context' is, of course, widely used to 
determine a specific interface with information or data. 
In archaeology, relativity, and so interpretation, is 
confirmed by the finds, i.e. artefacts, discovered within 
soil deposits or 'contexts'. An artefact can be one or all 
of the following: a clue, a treasure, an indicator of 
function or activity, or a linkage to the cultural, 
historical environment. Such a metaphorical tool would 
be a valuable addition to those presented in this paper, 


offering depth and elaboration within a sonic, narrative 
environment. 

5 CONCLUSION 

The reversal of archeological recording methods to 
assist the audio design process of a creative virtual or 
augmented environment has been presented in this 
paper through abstraction. Also, by touching upon 
issues of agency and the coded record, this investigation 
points towards theories that may further assist in the 
problematic issues of non-linear narrative. 

Experience of production infers that computer¬ 
generated stratigraphical sequences are a helpful tool 
assisting flexible creation and visualization of relativity 
between units of narrative content. By breaking down 
the content into different groupings or phases, 
sequences of events can be planned, using the metaphor 
of stratigraphy. This is imagined, without depth, in 
virtual/augmented environments as layers of successive 
present tense 'overlays' of sound occupying the site. 
Hence, narrative development can usefully be managed 
by assigning sounds to exist in a particular phase. Each 
phase of the soundscape may contain the basic units of 
sonic narrative, namely consequential sounds, which 
reward active agency or movement, and consecutive 
sounds, which reward passive agency or standing still 
and listening. 

Likewise, the multi-functional classes presented here 
fulfill the purpose of translating these relations in the 
terms of agency, although 'real' time remains to be 
addressed satisfactorily. It is recognised that further 
temporal relations have to be identified beyond the 
present approach that accounts for sounds two nodes of 
existence - the beginning and end. The lack of 
malleability of the record during run-time, indicates 
next step research is necessary to consider 
programmable behaviours or AI attributed to the sound 
actor, and how such features would develop narrative 
potential. 

However, the overriding opinion gathered from the 
investigation is that, although the metaphor of 
archaeological record should be treated as a state of 
transference and modification, rather than empirical and 
static, it represents a structure of narrative potential that 
can be released by the interactor's agency. 
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